1. Field of the Invention
The present invention relates to a technique of providing a composite image obtained by compositing a real space image and a virtual space image.
2. Description of the Related Art
Mixed reality, i.e., so-called MR is recently known as a technology of merging a real world and a virtual world seamlessly in real time. There is an MR technology using a video see-through HMD (Head Mounted Display). An object almost matching an object observed from the pupil position of an HMD wearer is sensed by, e.g., a video camera. A CG (Computer Graphics) image is superimposed on the sensed image to generate a mixed reality image. The mixed reality image is provided to the HMD wearer.
FIG. 29 is a block diagram showing the functional arrangement of a general mixed reality system using a video see-through HMD. As shown in FIG. 29, the system includes an image processing apparatus 3002 and a video see-through HMD 3001.
As shown in FIG. 29, the video see-through HMD 3001 includes an image sensing unit 3003 that senses the outside world, a display unit 3004 that displays an MR image obtained by compositing a real space image sensed by the image sensing unit 3003 with a CG image, a three-dimensional position and orientation sensor unit 3005 to calculate the position and orientation of the viewpoint of the video see-through HMD 3001, and an I/F 3006 functioning as an interface for data communication with the image processing apparatus 3002.
The image processing apparatus 3002 generates a CG image on the basis of the sensed image and three-dimensional position and orientation information received from the video see-through HMD 3001 and generates a composite image by compositing the CG image with the sensed image. Generally, an apparatus such as a personal computer or workstation with an advanced calculation processing function and graphic display function is used as the image processing apparatus 3002.
The image sensing unit 3003 senses an observation image in the outside world which can be seen from a viewpoint almost matching that of the observer who is wearing the video see-through HMD 3001 on the head. More specifically, the image sensing unit 3003 includes two sets of image sensing elements and optical systems which respectively correspond to the right and left eyes and generate a stereoscopic image, and a DSP that executes an image process of the succeeding stage.
The display unit 3004 displays the composite image of the sensed image and CG image. The display unit 3004 also includes two sets of display devices and optical systems for the right and left eyes. A small liquid crystal device or a retina scan type device by MEMS is used as the display device.
The three-dimensional position and orientation sensor unit 3005 obtains the position and orientation information of the viewpoint of the observer. The three-dimensional position and orientation sensor unit 3005 uses a magnetic sensor or a gyro sensor (acceleration and angular velocity).
The I/F 3006 transmits a real space image sensed by-the image sensing unit 3003 and position and orientation information measured by the three-dimensional position and orientation sensor unit 3005 to the image processing apparatus 3002 and receives a composite image generated on the side of the image processing apparatus 3002. The I/F 3006 that is required to transmit an enormous quantity of data in real time uses a metal line such as a USB or IEEE1394 or an optical fiber such as GigabitEthernet.
The components of the image processing apparatus 3002 will be described next.
An I/F 3007 on the side of the image processing apparatus 3002 functions as an interface for data communication with the video see-through HMD 3001.
A position and orientation information generation unit 3008 obtains the position and orientation of the viewpoint of the observer. To do this, the position and orientation are obtained by using the position and orientation information received from the three-dimensional position and orientation sensor unit 3005. In addition, a marker in the sensed image from the video see-through HMD 3001 is extracted, and the position and orientation are corrected on the basis of the marker position.
A content DB (database) 3009 holds data to be used to display each virtual object included in a virtual space. A CG rendering unit 3010 forms a virtual space by using the data stored in the content DB 3009 and generates a virtual space image (CG image) observed from the viewpoint of the position and orientation based on the position and orientation information generated by the position and orientation information generation unit 3008.
An image composition unit 3011 generates a composite image (MR image) by compositing a CG image generated by the CG rendering unit 3010 with a sensed image received from the video see-through HMD 3001 and sends the composite image to the video see-through HMD 3001 via the I/F 3007. The display unit 3004 displays the received composite image, as described above.
With the above-described arrangement and process, the observer who wears the video see-through HMD 3001 on the head can experience, before his/her eyes, a mixed reality space obtained by merging a real space and a virtual space seamlessly in real time.
Japanese Patent Laid-Open No. 11-88913 discloses a general MR technology and system configuration (FIGS. 7A and 7B; paragraph No. 0035).
However, the above-described prior art has the following problems.
To obtain a more real MR space, the resolution and tonality of the sensed image and CG image (virtual image) must be high. To display a more smooth moving image, a higher frame rate is necessary. To meet these requirements, the data amount of a sensed image and MR image dramatically increases. The transmission band to transmit/receive the data is also required to be very wide. Especially when data transmission is done by wireless communication, the transmission band is limited as compared to a wired system. It is therefore very difficult to simultaneously ensure a high resolution, a high tonality, and a high frame rate for moving image display and real-time image display with a minimum delay time.
Image processing for accurate registration is executed only on the side of the image processing apparatus, i.e., one of the apparatuses included in the system. For this reason, the load is heavy. The time required for information processing for registration mainly executed by software is also increasing. This may hinder implementation of MR that should minimize the delay.